Papers by Lisa Anne Hendricks
Decoupling the Role of Data, Attention, and Losses in Multimodal Transformers (2021.tacl-1)
Copied to clipboard
| Challenge: | Recent studies suggest multimodal transformer models learn rich visual-linguistic representations. |
| Approach: | They focus on dataset noise and language similarity to their downstream task . they find that models with a multimodal attention mechanism outperform deeper models with modality-specific attention mechanisms. |
| Outcome: | The proposed models outperform models with a multimodal attention mechanism on downstream tasks. |
CulturalFrames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metrics (2025.findings-emnlp)
Copied to clipboard
Shravan Nayak, Mehar Bhatia, Xiaofeng Zhang, Verena Rieser, Lisa Anne Hendricks, Sjoerd Van Steenkiste, Yash Goyal, Karolina Stanczak, Aishwarya Agrawal
| Challenge: | CulturalFrames is a benchmark designed for rigorous human evaluation of cultural representation in visual generations. |
| Approach: | They propose to quantify the alignment of T2I models and evaluation metrics with respect to both explicit (stated) and implicit (unstated, implied by the prompt’s cultural context) cultural expectations. |
| Outcome: | The proposed model is based on 983 prompts, 3637 images and 10k human annotations from 10 countries and 5 socio-cultural domains. |
Measuring Progress in Fine-grained Vision-and-Language Understanding (2023.acl-long)
Copied to clipboard
| Challenge: | X-VLM models lack "fine-grained" understanding of relationships, verbs and numbers in images . pretraining on large-scale image–text data from the Web has facilitated rapid progress on many vision-and-language tasks . |
| Approach: | They investigate models that outperform other baselines on fine-grained data . they highlight importance of novel losses and rich data sources for learning fine-grain skills . |
| Outcome: | The proposed model outperforms baseline models on four fine-grained benchmarks . the model outpersforms other baseline models and even degrades performance . |
Localizing Moments in Video with Temporal Language (D18-1)
Copied to clipboard
| Challenge: | a novel model for localizing moments in a longer video using natural language queries is challenging . moment localization is similar to other language and vision tasks, but it offers an interesting opportunity to model temporal dependencies and reasoning in text. |
| Approach: | They propose a model that explicitly reasons about different temporal segments in a video . their dataset includes a dataset with real videos and template sentences . |
| Outcome: | The proposed model explicitly reasons about different temporal segments in a video . it shows that temporal context is important for localizing phrases which include temporal language . |
Challenges in Detoxifying Language Models (2021.findings-emnlp)
Copied to clipboard
Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, Po-Sen Huang
| Challenge: | Prior work often relies on automatic evaluation of LM toxicity. |
| Approach: | They evaluate toxicity mitigation strategies for automated and human evaluations . they find human raters disagree with high automatic toxicity scores after strong toxicity reduction interventions . |
| Outcome: | The proposed methods reduce LM toxicity but lower coverage for marginalized texts . human raters disagree with high toxicity scores after strong toxicity reduction interventions . |
Object Hallucination in Image Captioning (D18-1)
Copied to clipboard
| Challenge: | Existing image captioning metrics do not capture image relevance . current metrics only measure similarity to ground truth captions . |
| Approach: | They propose a new image relevance metric to evaluate captioning models with veridical visual labels and assess their rate of object hallucination. |
| Outcome: | The proposed metrics show that models with veridical visual labels have higher hallucination rates than models with lower hallucinosity. |
Probing Image-Language Transformers for Verb Understanding (2021.findings-acl)
Copied to clipboard
| Challenge: | Multimodal image-language transformers have achieved impressive results on a variety of tasks that rely on fine-tuning. |
| Approach: | They collect a dataset of image-sentence pairs consisting of 421 verbs . they evaluate pretrained image-language transformers and find they fail more in situations that require verb understanding compared to other parts of speech. |
| Outcome: | The proposed model trains on a manually-annotated and smaller dataset does better on the task. |